test_models: fuzz TX messages against panda safety - #3723
Open
ping-dev-ui wants to merge 1 commit into
Open
Conversation
Contributor
Car behavior reportReplays driving segments through this PR and compares the behavior to master. ✅ No changes detected |
Replays route CAN into both the car interface and panda safety so they derive the same vehicle state, builds the CarControl controlsd would send for fuzzed controls, and asserts every message the car controller emits passes safety_tx_hook. Engagement is taken from panda's own controls_allowed rather than forced, so a rejection is a limits mismatch and never an artifact of the test making the two sides agree. Scenarios wind the rate limiters to the actuator bound, toggle engagement mid-command, and skip control frames so the real time rate checks see an irregular cadence. Sent messages are counted per address so failures say what was exercised. test_panda_safety_tx_edge covers the other direction: an accepted steering command re-packed with the signal at the DBC field's extreme must be rejected, with the unmodified re-pack asserted accepted first so a rejection cannot be a round-trip artifact. Also adds Fuzzy.real() to opendbc/testing.py. Closes commaai/openpilot#32425 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
ping-dev-ui
force-pushed
the
fuzz-tx-messages
branch
from
September 7, 2026 11:00
07999db to
0e4ba7c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TITLE: test_models: fuzz TX messages against panda safety (#32425)
Closes commaai/openpilot#32425 (bounty).
What this adds
Two tests in
test_models.py, run for every platform with a route:test_panda_safety_tx_fuzzyreplays route CAN into both the car interface and panda safety so both derive thesame vehicle state, then builds the
CarControlcontrolsd would send for fuzzed controls and asserts every messagethe car controller emits passes
safety_tx_hook. A rejection is a mismatch between the limits openpilot applies toits commands and the ones panda enforces on them.
Three things it does differently from a plain random walk:
enabledis taken from panda's owncontrols_allowed, derived from the replayedCAN exactly as on the car. The existing
test_panda_safety_carstatealready asserts panda's engagement agreeswith
CarState, so this test inherits that and a rejection can only be a limits mismatch, never an artifact ofthe test making the two sides agree.
hold_windupholds the actuator bound for half thewindow so the rate limiters wind fully up, then steps to zero;
churntoggles engagement while a command islive;
cadence_jitterdrops control frames so the real time rate checks see an irregular cadence.(address, bus)and included in the failure message, soa rejection says what was being exercised, and a silent "sent nothing" cannot pass.
test_panda_safety_tx_edgecovers the direction the fuzz test cannot: it takes an accepted steering command,decodes it, re-packs it unchanged (and asserts that is still accepted, so a rejection is never a round-trip
artifact), then re-packs it with the steering signal at the DBC field's own extreme and asserts panda rejects it.
openpilot never emits such a frame, so an accept-only fuzzer would stay green if panda stopped enforcing a cap.
Table-driven per brand (
STEER_CMD_SIGNALS), with a candidate list per entry because a brand can span CAN generationsthat send different messages (Hyundai:
LKAS11on CAN,LKAS/LKAS_ALT/LFAon CAN FD, chosen by flags in thecar controller); the first accepted frame whose address matches a candidate is the one probed. Brands or platforms
with no matching candidate skip.
Also:
Fuzzy.real()inopendbc/testing.py, same edge strategy asinteger().A note on
mutation.py, because I got this wrong in the first push and CI caught it. I had deleted theknown_survivorsallowlist after a local mutation run reported 3020/3020 killed. CI then reported exactly thosethree surviving. The safety suite includes fuzzy tests seeded per process, so a lucky seed can fail under any mutant
and my local "kill" was noise. Line 188 is in fact an equivalent mutant:
250001U / 2Uis still125000underunsigned integer division, so no test can ever distinguish it. The allowlist is restored unchanged in this PR. The
two sign-boundary mutants at 218/219 (
> 0to> 1,>= 0to>= 1) look deterministically killable with aprobe at
desired_angle_lastof exactly one CAN unit; I would rather do that as a separate small PR againstcommon.pythan widen this one.Results
Both tests run on every platform with a route. Locally (WSL, host-side libsafety), on thirteen platforms chosen to cover
every steering type and both longitudinal modes:
tx_fuzzy(25 examples)tx_edgeLKAS11)LKAS)LKAS_ALT/LFA)Zero rejections from the fuzz test on every platform that ran it. The full
test_modelssuite withMAX_EXAMPLES=1, as CI runs it: 2610 tests, 0 failures, 376 skipped (the pre-existing dashcamOnly, SecOC and notCar skips), 379 s on 24 workers.No new mismatches surfaced on these ten. That is a real result, not a disappointment: I would rather report that
the current limits agree on the routes tested than tune scenarios until something fails. The scenarios and the
edge probe are the coverage that was missing; the table for
tx_edgeis easy to extend brand by brand.Runtime
Slowest platform seen (TOYOTA_RAV4_TSS2_2023), single run each:
tx_fuzzy,MAX_EXAMPLES=25(default)tx_fuzzy,MAX_EXAMPLES=1(CI)tx_edgecarstate_fuzzy, 300 examples (reference)About 5 s of each number is fixed import and setup cost. At CI's
MAX_EXAMPLES=1the two tests together cost roughlythe same as the existing fuzz test; at the default 25 examples the fuzzer is about twice it. Route replay is the
price of catching the panda#1948 class, which a synthetic fuzzer cannot reach.
MAX_EXAMPLESis honoured as in the other fuzzy tests (CI runs withMAX_EXAMPLES=1).Notes
sample_t's are used panda#1948 (the torquewind-down using a sample minimum); a synthetic fuzzer without real CAN history cannot reach it. The replay
structure here follows the approach in test_models: fuzz the messages openpilot sends #3701, which I read before writing this; the engagement handling, the
scenarios, the coverage accounting and the edge test are new.
libsafety, the same way the other panda safety testsin this file run.